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DESCRIPTION 
AUTOFOCUS SYSTEM FOR SCANNING MICROSCOPY 

MICROFICHE APPENDIX 

A Microfiche Appendix containing computer source code is attached. The 
5 Microfiche Appendix comprises one (1) sheet of microfiche having 22 frames, 
including one title frame. 

The Microfiche Appendix contains material which is subject to copyright 
protection. The copyright owner has no objection to the reproduction of such 
material, as it appears in the files of the Patent and Trademark Office, but 
10 otherwise reserves all copyright rights whatsoever. 

TECHNICAL FIELD 

The present invention relates to autofocusing and, more particularly, to a 
system for microscope autofocusing. 

BACKGROUND ART 

15 Autofocus is a requirement for any fully automated microscope-based 

image processing system that must scan areas larger than a single field. 
Experience has shown that it is not possible to maintain focus simply by 
determining the best foci at two points on a microscope slide and scanning along 
the line between them in three dimensional space. This may be due to many 

20 causes, including mechanical instability of the microscope and irregularity of glass 
slide surfaces. For example, thermal expansion could account for several microns 
of instability in microscopes with lamps acting as unevenly distributed heat 
sources. Using the coefficient of thermal expansion for aluminum, a 1.0°C 
increase causes 0.6 micrometer Ozm) of expansion for each 25 millimeters (mm) 

25 length between the objective and stage in a microscope. Mechanical instability 
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may also arise from gear slippage and settling between moving components in the 
stage. Microscope slide surface irregularity is another source of error. Standard 
optical quality mirror flatness is about 1.5 /un over 25 mm. Given that mirrors 
are ground glass and microscope slides are float glass, microscope slide surface 
5 irregularity could be much greater. According to the definition by others, such as 
Fran?on (Fran9on M: Progress in Microscopy. Row, Peterson, Evanston, 111., 
1961), the theoretical microscope depth of field for an objective with numerical 
aperture (NA) 0.75 is 0.74 /miata wavelength of 500 nm. Best focus can vary 
through a range of about 25 /nn in a horizontal scan of 50 mm across a 
10 microscope slide. Whatever the source of instability, autofocus can compensate 
given that the positional variations have relatively long time constants. 

Most autofocus methods fall into two categories: position sensing and 
image content analysis. Position sensing methods, such as interferometry, require 
independent calibration of the best focus location and, more importantly, a single 
15 well-defined surface from which to reflect light or sound. In light microscopy 
there are often two reflective surfaces, the coverslip and slide. In addition, tissue 
specimens can have significant depth and best focus is not necessarily achieved at 
the surface of the glass. These problems make absolute position sensing methods 
impractical for use in light microscopy. Image content analysis functions, such as 
20 used by the present invention for autofocusing the microscope, on the other hand, 
depend only on characteristics measured directly from the image. Best focus is 
found by comparison of these characteristics in a series of images acquired at 
different vertical positions. This method of autofocus requires no independent 
reference and is not affected significantly by the second reflective surface. Its 
25 most important limitation is speed, which is dependent on the video rate, the 
vertical repositioning time, function calculation time and search range. 

Image content autofocus functions have previously been compared for 
brightfield microscopy, but apparently not for fluorescence or phase-contrast 
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microscopy. For example, Groen, Young and Ligthart (Groen FCA, Young IT, 
Ligthart G: A comparison of different focus functions for use in autofocus 
algoritbms. Cytometry 6:81-91, 1985) compared 11 autofocus functions under 
brightfield using an electron microscope grid and a metaphase spread, and Vollath 
5 (Vollath D: Automatic Focusing by Correlative Methods. J Microsc 147:279-288, 
1987) tested an autocorrelation function under brightfield using a pearlitic steel 
specimen. Groen et al. concluded that three autofocus functions, i.e., two gradient 
functions and the intensity variance, performed the best. However, some autofocus 
functions that performed well on one specimen did not perform well on others and 
1 0 the authors cautioned against extrapolating the results to other imaging modes and 
specimens. 

The uncertainty in applying autofocus test results from one microscope 
method to another led to the present invention. The development of the present 
invention included exploring autofocus performance in microscopy of fluorescent 

15 stained biologic specimens. The fluorescent signal can be used directly for 
autofocus. However, problems summarized by others, such as Chen (Chen LB: 
Fluorescent labeling of mitochondria, in Fluorescence Microscopy of Living Cells 
in Culture, Part A, Wang YL and Taylor DL, eds. Academic Press, San Diego, 
103-123, 1989), including photobleaching and the formation of free radicals, 

20 singlet oxygen, and heat, can create conditions under which minimizing fluorescent 
excitation becomes critical. The most critical conditions probably occur in 
analyzing live cells. If the signal is weak and antiphotobleaching agents cannot 
be used because of toxicity, the signal could easily be completely lost in the 5 - 10 
video frames of exposure required for autofocus. In addition, the fluorescence 

25 byproducts themselves are toxic, and excessive exposure could alter the results or 
damage living cells. Therefore it is desirable to find a nondestructive imaging 
technique for autofocus. With brightfield microscopy, fluorescent stained cells 
appear unstained, showing very little contrast. Phase-contrast microscopy, on the 
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other hand, gives high contrast images of unstained cells and is more useful for 
autofocus. For these reasons, autofocus function performance was tested for both 
phase contrast and fluorescence microscopy. More details of different approaches 
for autofocus can be found in the doctoral dissertation of Jeffrey H. Price entitled 
5 Scanning Cytometry for Cell Monolayers, University of California, San Diego, 
1990, which is hereby incorporated by reference. 

DISCLOSURE OF INVENTION 

The present solution to the problem of fast and reliable autofocus of 
cellular components from photosensitive specimens and live cells in a system 
10 capable of scanning multiple microscope fields is the instant autofocus system for 
scanning microscopy designed to automate, simplify, accelerate, and improve the 
quality of the process. The goal of the autofocus system is to accurately and 
automatically position the focus positioner, or focus mechanism, of the microscope 
so as to gather information and present it for further processing. 

BRIEF DESCRIPTION OF DRAWING 

The objects, advantages and features of this invention will be more readily 
appreciated from the following detailed description, when read in conjunction with 
the accompanying drawing, in which: 

Fig. 1 is a high-level diagram illustrating the basic components of a 
presently preferred embodiment of the autofocus system of the present invention; 

Figure 2 is a representation of a magnified image of cells as seen through 
the microscope of the system shown in Figure I; 

Figure 3 is a block diagram of the presently preferred image processor of 
Figure 1; 

Figure 4 is a high-level flow diagram of an autofocus process performed 
by the system shown in Figure 1; 



15 



20 



25 



W 96/01438 PCTAJS95/08424 



5 

Figure 5 is a flow diagram of a binary search autofocus process performed 
by the system shown in Figure 1; 

Figure 6 is a flow diagram of a sequential autofocus process performed by 
the system shown in Figure 1; 
5 Figures 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i illustrate autofocus function 

(F,-F n ) results for a microscope field containing 10 fluorescent stained cells using 
the system shown in Figure 1; 

Figures 8a, 8b, 8c, 8d, 8e, 8f, 8g, 8h, and 8i illustrate autofocus function 
(F,-F n ) results for a phase contrast microscope field containing 10 cells using the 
10 system shown in Figure 1; 

Figures 9a, 9b, 9c, 9d, 9e, 9f, 9g, 9h, and 9i illustrate autofocus function 
(F r F n ) results for a microscope field containing a single fluorescent stained cell 
using the system shown in Figure 1; 

Figures 10a, 10b, 10c, 10d, lOe, lOf, lOg, 10h, and lOi illustrate autofocus 
15 function (F,-F n ) results for a phase contrast microscope field containing a single 
cell using the system shown in Figure 1; 

Figure 1 1 is a three-dimensional plot of the response of function F 3 against 
focus position and zoom using the system shown in Figure 1; and 

Figure 12 is a chart showing experimental results in autofocus precision, 
20 accuracy and speed for the phase contrast and fluorescence methods. 

Figure 13 is a microscopic system schematic that illustrates the operation 
of an autofocus system using volume imaging according to the invention; 

Figures 14A and 14B are block diagrams of a first embodiment of a 
volume imaging apparatus; 
25 Figures 15A and 15B are diagrams of a second embodiment of a volume 

imaging apparatus; 

Figure 1 6 illustrates a microscope field being scanned by a volume imaging 
array according to the invention; 



WO 96/01438 



PCT/US95/08424 



6 

Figure 17 is an illustration of a buffer for image alignment in the volume 
imaging apparatus; 

Figure 18 is a schematic representation of a volume image stored in a 

buffer of Figure 17; 

5 Figure 19 is a flow diagram illustrating the autofocus procedure using the 

volume imaging apparatus; 

Figure 20 is a plot showing a optical quality metric for best focus selection 
using volume imaging; and 

Figure 21 is an embodiment of a volume imaging autofocus system using 

10 optical highpass filtration. 

BEST MODES FOR CARRYING OUT THE INVENTION 

AUTOFOCUS BY VERTICAL POSITIONING 

The following detailed description of preferred embodiments of 
autofocusing by vertical repositioning presents a description of certain specific 
1 5 embodiments to assist in understanding the claims. However, the present invention 
can be embodied in a multitude of different ways as defined and covered by the 
claims. 

For convenience, the following description is topicalized into the following 
principal sections: I. Materials and Methods, U. Metrics for Performance: 
20 Autofocus Functions, III. Performance Results, and IV. Conclusions. A more 
detailed outline of the description is as follows: 

I. MATERIALS AND METHODS 

A. Microscope and Image Processor Overview 

B. Microscope and Video Camera 
25 C. Positioners 



WO 96/01438 PCT/US95/08424 



7 



D. Lamps and Exposure Control 

E. Image Processor, Computer and Software 

F. Cells and Specimen Preparation 

G. Basis for Comparison of Autofocus Functions 
5 H. General Autofocus Process 

I. Binary Search Autofocus Process 
J. Sequential Autofocus Process 

K. Automated Scanning and Real-Time Focus Calculation 

II. METRICS FOR PERFORMANCE: AUTOFOCUS FUNCTIONS 
10 A. Functions Based on Resolution 

B. Functions Based on Contrast 

C. Functions Based on Combined Resolution and Contrast 

D. Functions Based on Autocorrelation 

III. PERFORMANCE RESULTS 

1 5 A - Evaluation of Autofocus Functions on Selected Microscope Fields 

1. Microscope Field with Ten Cells 

2. Microscope Field with One Cell 

3. Function Dependence on Magnification and Sampling 
B. Autofocus Performance in Automated Scanning 

20 1- Accuracy, Precision and Speed 

2. Phase Contrast Focus as an Estimate of Fluorescence Focus 

IV. CONCLUSIONS 

I. MATERIALS AND MFTHnng 

A. Microscope and Image Processor Overview 
25 Figure 1 illustrates the presently preferred embodiment of an autofocus 

system 100 of the present invention which uses vertical positioning to acquire 
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focus information. The hardware components of the system 100 include an 
epifluorescent microscope 102, a motorized stage 103, controlled by a pair of XY 
motors 104a and a Z motor 104b, XYZ stage controller 106, a piezoelectric 
positioner 107, a video camera 108, an image processor 1 10, and a host processor 
5 1 12. These components are further described below. 

B. Microscope and Video Camera 

In one presently preferred embodiment, the cells (Figure 2) are imaged on 
a Nikon Optiphot 102 (Figure 1) through a CF Fluor DL 20x C, 0.75 NA 
objective with Ph3 phase contrast. This fluorite objective provides high UV 

10 transmission. The epifluorescence filter cube has a 365 nm ± 10 nm (50% of 
peak) bandpass excitation filter, a 400 nm dichroic mirror and no barrier filter. 
In experiments, the images were further magnified through a Nikon CCTV 0.9 - 
2.25 zoom lens onto a Dage VE 1000 RS-170 CCD camera 108. Experiments 
were performed at a zoom of 1.0 except for the sampling experiments, which were 

15 carried out at a series of magnifications. For phase contrast, a Nikon 0.52 NA 
long working distance condenser is used. 

C. Positioners 

The microscope stage 103 (Figure 1) is moved laterally under computer 
control by stepper motors. The stage 103 is built by Syn-Optics (Sunnyvale, CA) 

20 and modified by New England Affiliated Technologies (Lawrence, MA) for finer 
stepping and simpler computer control. The smallest step size is 0.127 fim. The 
stage 103 is controlled by a New England Affiliated Technologies 103M 
microstepping driver and an Oregon Micro Systems, Inc. (Beaverton, OR) PCX 
AT ISA-bus compatible computer board. 

25 Focus is changed with a piezoelectric objective positioner ("PIFOC") 107 

and an E-810.10 closed loop controller (Polytech PI, Costa Mesa, CA). The piezo 
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positioner 107 is sandwiched between the objective turret and the objective of the 
microscope 102. Measurements with an oscilloscope reading the built-in linear 
variable differential transformer (LVDT) sensor output showed that movements of 
< 1 (jm occurred in < 10 milliseconds (ms) with the fluorite objective, and 
5 response was dependent on objective mass. To retain the 160 mm tube length of 
the Optiphot 102, the objective turret is replaced by a custom-machined adapter. 
The 13 mm thick objective positioner significantly reduces image quality if this 
is not done, but movement through the 100 fim (0.004°) range does not 
measurably degrade the image. Position is controlled by output from a digital-to- 

10 analog converter in a Keithley Metrabyte (Taunton, MA) DAS- 1600 Data 
Acquisition Board. The 12-bit D/A converter divides the 100 /im range of the 
PIFOC 107 into 4096 steps of 24 nm each. Due to the previously discussed 
temperature and mechanical instabilities of the microscope 102 itself, actual focus 
accuracy is not better than a few microns over long periods, but for the required 

1 5 focus interval of a fraction of a second, the precision approaches the minimum step 
size. 

D. Lamps and Exposure Control 

For fluorescent autofocus tests, specimen exposure is controlled with a 
Uniblitz Model D122 Driver and Shutter (Vincent Associates, Rochester, NY). 

20 The fluorescence lamp is an Osram lOOw HBO W/2 mercury vapor arc lamp in 
a Nikon HMX-2 lamp house. Variability of < ±3% over 3 hours with this lamp 
is measured by illumination of the cell stain solution described above, modified by 
an addition of 10 ftg/ml DAPI and 1 mg/ml Herring Sperm DNA (Sigma, St. 
Louis). This solution is placed in an acrylic well under a sealed coverslip. For 

25 phase contrast, exposure is controlled with an EG&G Electro-Optics PS 450AC 
Power Supply (Salem, MA) and an XSA 80-35S-30171 xenon flash lamp 
(Advanced Radiation Corp., Santa Clara, CA). A Nikon HMX-2 lamp house was 



WO 96/01438 



PCT/US95/08424 



10 

modified to house the xenon flash lamp and wired to the 450AC power supply. 
The strobe is triggered by the timer circuit on the data acquisition board. The 
timing for the strobe is supplied by a vertical blank hardware interrupt from the 
image processor 1 10 (Figure 1). The data acquisition board has a programmable 
5 strobe delay that is set for 14 ms to assure that the objective positioner has 
completed movement prior to image acquisition. The strobe rate is 60 Hertz (Hz) 
during phase contrast focus testing. The average stability of this lamp is better 
than the mercury vapor arc lamp, but there are occasional intensity spikes. 

To perform an autofocus calculation in real time, either the focus position 
10 has to be moved in less than 16 ms (for 60 Hz operation), or the position has to 
be moved at a constant velocity and the image frozen with a strobe. For the best 
performance with the incremental movement utilized herein, the image is collected 
after movement has been completed. This is done in phase contrast by delaying 
the strobe 14 ms after the vertical blank in the video signal (when the command 

15 to change position is sent to the PIFOC 107). This delay insures that the focus 
position has changed before the image is collected by the video camera 108. The 
14 ms delay and the strobe would not be required if the position could be changed 
during the vertical blank interval of about one ms. Better feedback control 
electronics on the PIFOC 107 would allow movement to occur fast enough to 

20 eliminate the need for the strobe. 

E. Image Processor. Computer and Software 

An Imaging Technology, Inc. Series 151 Image Processor 1 10 (Figure 1) 
is used for speeding image operations. A block diagram of the preferred image 
processor 1 10 is illustrated in Figure 3. It should be observed that while an image 
25 processor 110 will generally speed up the autofocus operation of the present 
invention, if speed is not critical there is no reason why the calculations performed 
therein could not take place in the host processor 112 (Figure 1) or any other 
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computer. The image processor 1 10 is preferably configured with six functional 
units, or boards, as follows: 1) a 512 x 512 8-bit Variable Scan Interface 130 for 
analog to digital conversion of the video signal generated by the camera 108, 2) 
a 512 x 512 x 32-bit Frame Buffer 132 for storage of four digital images, 3) a 
5 1024 x 1024 x 32-bit Frame Buffer (not shown, but similar connections are made 
to the buses of the processor 1 10 as shown for the frame buffer 132) for storage 
of sixteen digital images, 4) a Histogram/Feature Extractor 134 for creating a 10- 
bit intensity histogram, 5) a Real Time Modular Processor (RTMP) 138 with a 
Real Time Sobel module for 8 x 8 convolutions and a 16-bit look-up-table, and 

10 6) an Arithmetic/Logic Unit 136 for multiplication, subtraction, addition and 
scaling. The RTMP 138 is a single board with three plug-in connections for sub- 
modules. The Real Time Sobel module utilizes two of these connections and the 
look-up-table utilizes the third connection. All of these operations proceed at 
video rates and can be pipelined for parallel operation. 

1 5 The key components of this system for testing the autofocus functions are 

the 8 x 8 convolver (part of RTMP 138) and the histogrammer 134. For most of 
the autofocus functions, the image is convolved and then histogrammed in a single 
pipelined video frame or field. The histogram is used to calculate the intensity 
sum, sum of squares and statistics, e.g., variance or standard deviation, with 

20 filtered image results truncated to 8 or 10 bits/pixel. The calculation results are 
further used to calculate a measure of focus. For 16-bit calculations, the image is 
first transferred to the host computer. Small differences are sometimes observed 
between the 8-bit and 10-bit results, but no further improvement is observed 
utilizing 16-bit results. Therefore, only 10-bit data for autofocus of selected fields 

25 is reported. For the presently preferred 60-Hz scanning implementation, the 
absolute value of the filtered images is taken prior to truncation to 8 bits. The 
host computer 1 12 (Figure 1) is preferably an AT-compatible 33 megaHertz (MHz) 
Intel i486 personal computer (PC). 
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The software programs to implement the autofocus process (Figure 4) and 
related control functions are written in *C and assembler. A portion of the *C 
and assembler source code is included in the attached Microfiche Appendix. The 
C routines are compiled with Metaware High C (Santa Cruz, CA). A Phar Lap 
5 (Cambridge, MA) assembler is used for the interrupt service routines that are 
running in the background. All object code is linked with the Phar Lap 386 DOS 
Extender. The Imaging Technology Series 151 C Library source code is also 
ported to ANSI C and recompiled with Metaware High C. This combination 
allows use of the full 32-bit capability of the i486 CPU by programs running under 
10 16-bit DOS. 

F. Cells and Specimen Preparation 

A portion of an example specimen, such as the specimen 1 14 of Figure 1, 
is shown in Figure 2. Figure 2 represents a magnified image of a typical specimen 
comprising a set of cells, particularly cell nuclei, generally indicated at 116. 

1 5 In one experiment, NIH 3T3 cells were plated on washed, autoclaved #1 .5 

coverslips. The cells were maintained in Eagle's minimal essential medium with 
Earle's salts, supplemented with 10% fetal bovine serum, 100 /xg/ml gentamicin, 
and 0.26 mg/ml L-glutamine (final concentrations), in a humidified 5% C0 2 
incubator at 37°C. After 1 day of cell growth, the coverslips were washed in 

20 phosphate buffered saline (PBS), fixed for 10 minutes in 4% paraformaldehyde in 
60% PBS, and stained for one hour. The stain solution consisted of 50 ng/ml 4\6- 
diamidino-2-phenylindole dihydrochloride (DAPI, Molecular Probes, Eugene, OR), 
10 mM THIS, 10 mM EDTA, 100 mM NaCl, and 2% 2-mercaptoethanol as 
described by others, such as Hamada and Fujita (Hamada S, Fujita S: DAPI 

25 Staining Improved for Quantitative Cytofluorometry. Histochem 79:219-226, 
1983). After staining, a few drops of DAPI solution were placed on a glass slide, 
the coverslips were laid face down over the solution, excess solution was wicked 
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away with tissue, and the coverslips were sealed to the slide with nail polish. This 
stain solution was found to exhibit excellent antiphotobleaching properties. 
Although photobleaching was avoided with this preparation, the degree of 
photobleaching can vary markedly with different techniques. This specimen also 
5 did not exhibit significant autofluorescence, which if nonspecific and diffuse could 
degrade performance by reducing contrast. 

G. Basis for Comparison of Autofocus Functions 

There is no independent standard against which autofocus functions can be 
tested. Therefore, performance must be rated by comparison. Groen et al., loc. 

10 cit , suggests eight criteria for comparing the performance of autofocus functions. 
These are: 1) unimodality, or the existence of a single maximum or minim jm^ 2) 
accuracy, or coincidence of the extremum and best focus; 3) reproducibility, or 
a sharp extremum; 4) range, or the vertical distance over which the function will 
unambiguously determine the direction to best focus; 5) general applicability, or 

15 the ability to work on different classes of images; 6) insensitivity to other 
parameters, or independence from influences such as changes in mean intensity; 
7) video signal compatibility, or the ability to use the same video signal as is 
utilized for image analysis; and 8) implementation, that is, it should be possible 
to calculate the function rapidly. 

20 The first three criteria - unimodality, accuracy and reproducibility - are 

most important for automated scanning. The range is less important because focus 
is usually performed on a field immediately adjacent to one where best focus was 
just calculated. Comparisons of microscope autofocus functions performed by 
Groen et al., loc. cit, led to the conclusion that the fifth criterion, general 

25 applicability for all types of images, cannot necessarily be expected. For a 
scanning system, however, it is sufficient to require applicability to one microscope 
imaging method (e.g., phase contrast or fluorescence) for all microscope fields. 
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The seventh criterion, video signal compatibility, is hardware dependent and is 
easily satisfied. The eighth criterion, implementation, is dependent on computer 
speed and function complexity. 

H. General Autofocus Process 
5 Referring to Figure 4, a high-level description of the autofocus process 200 

will now be given. Note that autofocus is controlled from the host processor 1 12 
(Figure 1). The host processor 1 12, or the image processor 110 under control of 
the host processor, can perform a transformation on the image and obtain a value 
which represents a degree of focus. This value can then be compared with another 

10 value obtained from another image after the stage 103 is moved up or down via 
the XYZ stage controller 106 to indicate the next direction of stage movement or 
after the objective is adjusted by the PIFOC 107. 

Beginning at a start state 202, the system 100 proceeds to a state 204 to 
acquire a digital image of the specimen 114 on the stage 103 at a first vertical 

15 position. The image is captured by the image processor 110. Moving to state 206, 
the system 100 applies a filter to the digital image to produce an intermediate or 
filtered image. The presently preferred embodiment utilizes a digital filter, and 
more specifically, the image sharpening transformation defined in function F 7 of 
Table 1. Of course, other filters, including analog filters can be used. Proceeding 

20 to state 208, the system 100 applies a measurement function to the intermediate 
image to obtain a magnitude. The presently preferred embodiment uses a contrast 
measurement function which utilizes the variance or standard deviation of image 
intensity, or the sum of the squares of the image intensity. Moving to a decision 
state 210, the system 100 determines if the Nth image has been acquired. The 

25 value of N utilized varies according to the specific autofocus method employed. 
For the purposes of this discussion, and as an example, N will be equal to two. 
Therefore, during the first pass of states 204 to 210, only the first image is 
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acquired, and the flow continues at state 212 wherein the stage 103 is moved by 
Z motor 104b in a vertical direction to a new (second) vertical position. In the 
preferred embodiment there are two focus mechanisms, therefore the piezoelectric 
PIFOC positioner 107 is moved instead of the stage 103 for fast autofocus, and 
5 both the PIFOC 107 and the stage 103 are to be moved together to combine fast 
autofocus and extended focus range. 

After the stage 103 has been moved to the new vertical position, or the 
PIFOC 107 adjusts the objective, the flow continues at state 204, wherein a digital 
image is acquired at the second vertical position. The states 206 and 208 are 
1 0 executed again using the second image to obtain a second magnitude. Moving to 
state 210, when the Nth image has been acquired, as for the current example of 
N=2, the system 100 proceeds to state 214. At state 214, the acquired magnitudes 
are compared and the focus is adjusted based on the results of the comparison. 
For the current example, if the magnitude at the first vertical position is greater 
15 than the magnitude at the second vertical position the focus mechanism is moved 
toward the first vertical position, else if the magnitude at the first position is less 
than the magnitude at the second position, the focus mechanism is moved toward 
the second vertical position. The PIFOC 107 and the vertical stage stepper motor 
positioner are both focus positioners. The PIFOC is much faster, but has a shorter 
20 range (100 pan or 0.004" in the presently preferred model, 200 /im in an alternate 
embodiment. The stepper motor, moving the entire mass of the stage, rather than 
just the objective, takes longer and cannot be use din a real time calculation, but 
has a range limited only by the room between the specimen and objective and the 
physical design of the microscope. Either or both can be used for autofocus. For 
25 a slow system with focus in a few seconds, the stage is fast enough. For a fast 
system requiring focus in a fiaction of a second, the PIFOC 107 is necessary. For 
applications requiring greater range and fast f cus, the stage could be focused first 
and all subsequent focusing done by the PIFOC 107 until the range is exceeded. 
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The stage could then be adjusted as necessary to keep the PIFOC 107 within its 
range. After the focus has been adjusted at state 214, the autofocus process 200 
completes at an end state 216. 

A number of different autofocus functions, which are further discussed 
5 below, may carry out one or both of states 206 and/or 208. 

The following two section will describe two specific methods utilized for 
autofocus: binary search and sequential autofocus. 

I. Binary Search Autofocus Process 

Referring to Figure 5, the binary search autofocus process 250 will now be 

10 described. This process is a more specific version of the process 200 shown in 
Figure 4. Autofocus process 250 uses the well-known binary search algorithm to 
move the stage 103 (Figure 1) and locate best focus. The search range is fixed as 
the distance of focus search interval and the center of the range is the center of 
focus search interval. Binary search autofocus is carried out by defining two focus 

15 positions between which focus is thought to exist and sequentially dividing the 
range in half to narrow down on best focus. The range is narrowed in this manner 
until it is smaller than the precision needed to identify best focus. 

The binary search autofocus process 250 begins at a start state 252, wherein 
a user, or an automated scanning program, requests focusing. The range is set to 

20 an initial value, e.g., 4 microns. The system 100 then moves to a state 254 
wherein the focus mechanism is positioned at the center of the focus test range. 
The focus mechanism is the PIFOC 107 or the stage 103 (note the vertical control 
is 104b which moves the stage 103 through a series of gears inside the 
microscope), or both. An image is then acquired, filtered, and histogrammed by 

25 using the image processor 110. From the histogram, a single focus function result 
(Fc) is calculated for that position and stored. Moving to state 256, the focus 
mechanism is positioned at the beginning of the range, and another image is 



96/01438 



PCT/US95/08424 



17 

acquired, filtered, and histogrammed. Another focus function result (Fa) is 
calculated and stored as before. Proceeding to state 258, the focus mechanism is 
positioned at the end of the range, and another image is acquired, filtered, and 
histogrammed. From the histogram, a single focus function result (Fb) is 
5 calculated for that position and stored as before. States 252 through 258 comprise 
an initialization sequence of the process. The rest of the flow chart states 
represents the loop that is executed to narrow the range until focus is found. 

Continuing at a decision state 260, a check (is range less than minimum) 
is made to see if the desired precision has been achieved. The minimum is user 
and application dependent. A practical example of the minimum is between 0.001 
fim and 1.0 fun depending on the demands of the application. The minimum step 
size of the presently preferred PIFOC 107 is 0.024 jim and is determined by the 
digital/analog converter control board in the host computer 1 12 and the electronics 
of the PIFOC controller. If the range is small enough focus has been located, and 
the flow completes at an end state 272. If the range is not less than minimum as 
determined at decision state 260, the system 1 00 continues at state 262 wherein the 
range is decreased by half. Proceeding to a decision state 264, a determination is 
made if the focus value result Fa is greater than the result Fb. If Fa, the focus 
value at the beginning of the range, is greater than Fb, the focus value at the end 
of the range, then the focus is closer to the beginning of the range. The new range 
will be defined by the old beginning and the center. If not, the focus is closer to 
the end of the range and the new range will be defined by the old center and end. 

If Fa is greater than Fb, the system moves to state 266, due to the focus 
value being closer to the beginning of the old range. The system 100 sets the new 
center to a position between the center and start of the old range, and moves the 
focus mechanism to the new center. The system also places the focus value (Fc) 
at the old center into the storage represented by Fb because this is the new end of 
the range. The system 100 further acquires, filters, histograms and calculates the 
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focus function value for the new image and stores the result in Fc. At the 
completion of state 266, the system 100 loops back to decision state 260 to 
determine if the range is now less than minimum, as previously described 

If Fa is not greater than Fb, as determined at decision state 264 and 
asserted at state 268, the system moves to state 270, due to the focus value being 
closer to the end of the old range. At state 270, the system 100 sets the new 
center to a position between the old center and old end of the range, and moves 
the focus mechanism to the new center. The system also places the focus value at 
the old center (Fc) into the storage represented by Fa because this is the new start 
of the range. The system 100 further acquires, filters, histograms and calculates 
the focus function value for the new image and stores the result in Fc. At the 
completion of state 270, the system 100 loops back to decision state 260, as 
previously described. 

J. Sequential Autofocus Process 

Referring to Figure 6, the sequential autofocus process 300 will now be 
described. The sequential autofocus process 300, which is a more specific version 
of the autofocus process 200 (Figure 4), begins at a start state 302, wherein a user, 
or an automated scanning program, requests focusing. The range is set to an initial 
value, e.g., 4 microns. The system 100 then moves to a state 304 wherein the start 
position is set to the initial value at the beginning of the range and the position 
number, n, is set to 0. Moving to state 306, the system 100 begins a loop (states 
306 through 310) that positions the focus mechanism and calculates the focus 
function at each position. The focus mechanism is positioned to the 'start position 
plus (n times the step size)*. The step size is the same as the "focus increment" 
in the experimental table of Figure 12. Focus increments of 0.102, 0.195, 0.244, 
0.146, 0220 and 0.073 micrometers were used in those experiments. Then the 
image is acquired, filtered, histogrammed and the focus function calculated. The 
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result is stored for later calculation of best focus. Proceeding to a decision state 
308, the system 100 determines if 'the current position minus the start position < 
range', i.e., whether the focus position sequence finished. If so, the system moves 
to state 312 to calculate best focus. If not, the system 100 moves to state 310 to 
increment the position number, n, by one, and continue moving and calculating the 
focus function results by looping back to state 306. 

If the focus position sequence is finished, as determined at decision state 
308, the system 100 moves to state 312 and computes the best focus. The power 
weighted average equation (Equation 1) is used to compute the best focus from the 
focus function values at positions 0 to z F 0 ...F 2 . At the completion of computing 
the best focus, the system advances to a decision state 314 to determine if the 
result lies too close to the beginning or end of the range. One example of too 
close is within 1/4 of the range. If the focus search range is 4 micrometers, and 
if the focus result were within 1 micrometer of either end of the range, then the 
center of the range would be repositioned and focus repeated. The actual value 
may vary depending on the application (specimen, required speed, etc.). If so, the 
true focus may lie outside the range. The system 100 proceeds to state 316 to 
change the center to the end of the range closest to calculated focus and repeat the 
focus sequence by looping back to state 304, as previously described. Note that 
the range stays the same, but the process succeeds as long as the focus values 
increase toward best focus. The range can be broadened automatically to speed 
finding best focus. This feature is important in an application where the focus at 
an adjacent field is not known, such as for a commercial microscope focus 
attachment that focuses regardless of the start position. Best focus is achieved if 
the result is close to the center, as determined at decision state 314, and therefore 
the system 100 proceeds to state 318 wherein the focus mechanism is positioned 
at best focus. The sequential autofocus process 300 completes at an end state 320. 
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K. Automated Scanning and Real-Time Focus Calculation 

One series of tests on the system 100 involved scanning areas of > 1000 
microscope fields in a raster pattern automatically. At each new field the best 
focus for the previous field is used for the center of the test focus range. The 
5 microscope is refocused at the beginning of the test sequence on each field until 
the calculated best focus fell into the inner half of the test range. This allows best 
focus to be achieved even if the best focus is outside the initial test range. In 
practice, the test range was wide enough to make refocusing rare. Before 
autofocus, the intensity of the fluorescence image is summed to verify the presence 
10 of at least one cell. If no cells are present, the field is skipped. At the beginning 
of a new row, the best focus from the beginning of the previous row is the center 
of the test range. Before the start of each experiment, a specimen is placed on the 
microscope, the scanning rectangle chosen, and the corners of the rectangle 
checked to verify that the foci are within the 100 jim range. At the first field, 

1 5 focus is performed manually. After the first field there is no human intervention 
until the scan is complete. 

Focus is calculated 20 times for both phase contrast and fluorescence on 
each field for statistical analysis of repeatability and comparison of accuracy. 
Precision is evaluated by the combined standard deviation of all focus trials from 

20 the entire scan. The combined standard deviation is computed by taking the square 
root of the average variance. Each focus test sequence is performed at 60 Hz by 
interrupt service routine control. An interrupt is initiated at the beginning of each 
vertical blank (60 Hz) by the Variable Scan Interface board on the Series 151 
Image Processor. The interrupt service routine controls the strobe, and accounts 

25 for the 2 vertical blank delay between image integration on the CCD chip 
(objective positioner movement) and histogram acquisition on the image processor. 
Accounting for the delay between positioning and image stabilization enables 
repositioning and measurement to occur at video rates. The path through the 
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image processor is from the digitized image in the Variable Scan Interface through 
the convolver to the histogrammer. The histogrammer input contains a 2-bank, 10- 
bit look-up table that is used to separate the odd and even fields for 60 Hz 
positioning and function calculation. Histogrammer look-up-table bank 0 is 
5 programmed to pass the even field unchanged and bank 1 is programmed to add 
256 to the odd field. The interrupt service routine switches banks on alternate 
vertical blanks. At the end of each odd field the interrupt service routine transfers 
the resulting 9-bit histogram and independently calculates the odd and even sum, 
sum of squares and pixel count. These values are placed in arrays accessible to 

10 'C routines for final calculation of the best focus position. The function results 
are also normalized by the number of pixels. 

After each focus sequence, with evaluation of the function at a number of 
positions, the maximum and the weighted average are used to find best focus. If 
the cells had been thinner than the depth of field of the microscope and the 

15 discrimination range of the focus function, the maximum would have been 
expected to perform well. In practice, however, the cells are thicker than the depth 
of field and much thicker than the discrimination range of the resolution functions 
(see Results section hereinbelow). Under these conditions, the function result is 
considered an estimate of the degree of focus at the corresponding position. A fit, 

20 or weighted average, of the data was performed during testing. Based on the ideal 
shape of the focus data, curve fits to a Gaussian and second and third order 
polynomials were tested. In each case, data were found with relatively aberrant 
shapes that caused the curve fits to perform very badly. The unusually shaped 
curves were probably produced by discrete distributions of cellular components in 

25 the vertical direction, causing a series of local maxima. For these reasons a 
weighted average of the form 
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Z 

where w a is the weight-averaged position, 2 is the vertical position (and to be 
distinguished from a specific function as such notation is used elsewhere herein), 
F is the result of a preselected autofocus function calculated from an image 
acquired at one position, and n is the power of the weighing, is used. The power 
5 accentuates the peak values. Over a narrow search range of 1 or 2 pm with 
increments significantly smaller than the depth of field, the function values are 
similar, and low powers result in best foci that are very close to the average 2. To 
improve sensitivity to the peak value, the power 'n' is increased to 4 and 8 before 
reasonable sensitivity to the maximum is achieved. These steps are shown in the 
1 0 flowchart of Figure 6 and are explained above. 

Sequential autofocus has an advantage over binary autofocus. Each focus 
position tested is defined before the focus routine begins. There is no dependence 
on the focus function value at previous positions. Therefore, delays, such as 
between integration of the image on the video camera 108 and digitization in the 
1 5 image processor 1 10, do not slow execution of the focus routine as much as they 
would with the binary search. 

In sequential autofocus, the focus position is moved to a series of locations, 
and at each location the image is acquired, fdtered, and histogrammed. Based on 
the histogram, a single focus value is calculated and stored. When the focus test 
20 sequence is complete, a power-weighted average of the focus function values is 
used to calculate best focus. Unlike binary autofocus, the calculated best focus 
position may lie in-between the tested positions. 

Both of autofocus methods assume that focus lies within a predetermined 
range. Most of the time, this is true in scanning microscopy because each new 
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field is directly adjacent to the previous one. If best focus lies at or near the ends 
of the range, the probability that h actually lies outside the range is higher. In the 
sequential autofocus method, if the best focus does lie close to the ends, the range 
is moved and autofocus process is repeated. 

5 n. METRICS FOR PERFORMANCE: AUTOFOCUS FUNCTIONS 

Image content autofocus functions are based on the assumptions that images 
increase in contrast and resolution (edge sharpness) as focus improves. In the 
contrast model, with an image that consists of light and dark regions, the light 
regions become darker and the dark regions become lighter as the equipment is 
moved farther from focus. This change in contrast can be described 
mathematically by the change in variance or standard deviation of pixel intensity. 
In the resolution model, detail blurs as the image moves out of focus. Resolution 
can be measured by analyzing the Fourier frequency spectrum or by the application 
of gradient, or highpass filters that isolate the high frequencies. The magnitude 
of the high frequencies or gradients can then be used as a measure of resolution, 
which is defined as a maximum at best focus. The effects of the defocusing on 
the optical transfer function have been discussed by others, such as Born and Wolf 
(Born M, Wolf E: Principles of Optics, 6th Edition, Pergamon Press, New York, 
1989), Erteza (Erteza A: Sharpness index and its application to focus control. 
Appl Opt, 15:877-881, 1976; Erteza A: Depth of convergence of a sharpness index 
autofocus system. Appl Opt 162273-2278, 1977), Goodman (Goodman JW: 
Introduction to Fourier Optics. McGraw-Hill, New York, 1968), Groen et al., loc. 
cit, and Hopkins (Hopkins HH: The frequency response of a defocused optical 
system. Proc Roy Soc A 231:91-103, 1955). Vollath, loc. cit., 1987, and (Vollath 
D: Verfahren und Einrichtung zur automatischen Scharfein-stellung eines jeden 
Punktes eines Bildes. German Patent DE 2,910,875 C 2, U.S. Patent 4,350,884, 
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1982, European Patent 0017726) derived additional autofocus functions based on 
autocorrelation and then suggested modifications for reducing the effects of noise. 

The eleven autofocus functions that were tested are summarized in Table 1 , 
along with references and calculation times on the computer hardware used here. 
5 The functions are divided into the following groups: 1) measures of resolution (F, 
- F 4 ), which are the sum of the squares of the result of a highpass filter, 2) 
measures of contrast (F 5 , F 6 ), represented by intensity variance or standard 
deviation, 3) combined measures of resolution and contrast (F 7 , F 8 ), and 4) 
autocorrelation functions (F 9 - F n ), which also incorporate components of 

10 resolution and/or contrast. The theory supporting the referenced functions is 
discussed by the respective authors. For mathematical description of these 
functions, the image is represented by g u where i and j are the spatial coordinates 
and g is the pixel intensity and all sums are double over i and j. In the equations, 
the dependence of the image and the autofocus function on vertical position is 

15 assumed (i.e., a function value is calculated from the image at each position). 
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Table I 
Aulofocus Functions 



Function 
1. Resolution 

F, = Z ([1 0 -1] * giJ ) 

\2 



*2 = 

F, = 



Fa' 



Z ([1 -i] * 
Z (H 2 -1] * Si) f 



zd 

9 



10 2. Contrast 



-1 -2 -1 
-2 12 -2 
-1 -2 -1 



5 n(n-iy t v f v 



3. Resolution and Contrast 



where = 



8 



-1 -1 -1 

-1 9 -1 
-1 -1 -1 



*« 



4. Autocorrelation 



Calculation Time* 
Frames/Position 



1,5* 
1,5^ 

4 

8 



* Convolution operation. 

n = i*j. the total number of pixels in the image. 

*RS-170 video format, V corresponds to calculation at the same rate as A/D conversion, 1 frame = 
33 ms, 1 field = 16 ms. 

**1 frame time if convolution resuh is absolute value truncated to 8 bits, and 5 frame times if 16- bit 
signed. 
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All of these functions utilize the entire image, rather than only the cells. 
In fluorescence, the cells, or objects, could be roughly identified using a threshold, 
and focus calculated from only the object pixels. This would add complexity, 
however, and slow autofocus. Since positioning is 2 fields ahead of function 
5 calculation, interposing the requirement to threshold the features and build a mask 
would cause a delay of at least a few frames. There could also be other problems. 
The distinct, bright nucleus becomes a dim, amorphous blur away from best focus. 
Even when this dim blur is barely discernible by eye, the direction to best focus 
can be easily determined by the algorithm used here. This is true well below the 
10 intensity where thresholding would yield a reliable object Also, the apparent 
object size increases out of focus, so thresholding at one position would not yield 
the same set of object pixels as thresholding at another. In phase contrast, 
segmentation of the cell features from image background would be much more 
difficult than a simple threshold, especially with the image out of focus. In 
15 preliminary experiments, it was observed that even with no cells in the field, the 
presence of small amounts of cellular debris was enough to keep the system in 
focus. This is probably because the large number of pixels dramatically improves 
the signal-to-noise ratio. For these reasons, it is advantageous to utilize the entire 
image for focus calculation. 
20 Independence from changes in illumination is desirable. Forphase contrast, 

the result at each position was divided by the mean or the square of the mean 
intensity, matching the order of the function dependence on intensity, to 
compensate for lamp fluctuations. For fluorescence, such a scaling is ill-behaved 
because mean intensity nears 0 not far from focus and it is a better measure of 
25 focus than of lamp intensity. Therefore, the functions were not similarly scaled 
for fluorescence autofocus. To correct for lamp fluctuations in fluorescence, 
independent measurement of lamp intensity are required. 
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The tested autofocus functions were chosen based on evaluations by other 
investigators and available computer hardware. Functions such as the thresholded 
absolute gradient and the thresholded video-signal content by Mendelsohn and 
Mayall (Mendelsohn ML, Mayall BH: Computer-oriented analysis of human 
5 chromosomes-III focus. Comput Biol Med 2:137-150, 1972) were not tested 
because performance was shown by Groen et al., loc. cit., to depend in part on the 
arbitrary choice of a threshold. The entropy function, such as described by 
Shannon (Shannon CE: A mathematical theory of communications. Bell Sys Tech 
J 27:379-423, 623-656, 1948) was shown by Firestone et al. (Firestone L, Cook 
10 K, Culp K, Talsania N, Preston K: Comparison of autofocus methods for 
automated microscopy. Cytometry 12:195-206, 1991) to be multimodal. Firestone 
et al., loc. cit., also tested the log spectral moment and two cellular logic functions 
that were not chosen because of hardware considerations. The log spectral 
moment requires the Fourier transform, which is still expensive to calculate at or 
1 5 near real time, and the cellular logic functions require different hardware than was 
available here for fast implementation. Variations of highpass filtering have also 
been implemented in analog circuitry by others, such as Dew, King and Mighdoll 
(Dew B, King T, Mighdoll D: An automatic microscope system for differential 
leukocyte counting. J Histochem Cytochem 22:685-696, 1974) and Johnson and 
20 Goforth (Johnson E, Goforth LJ: Metaphase spread detection and focus using 
closed circuit television. J Histochem Cytochem 22:536-545, 1974). 

A. Functions Based on Resolution 

Groen et al., loc. cit, reported F„ F 2 , and F 5 to be the best of 11 
functions tested for brightfield microscopy. F„ the squared gradient function 
25 described by others, such as Brenner et al. (Brenner JF, Dew BS, Horton JB, King 
T, Neurath PW, Selles WD: An automated microscope for cytologic research. J 
Histochem Cytochem 24:100-111, 1976), Erteza, loc. cit, and Mulier and 
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Buffington (Muller RA, Buffington A: Real-Time Correction of Atmospherically 
Degraded Telescope Images Through Image Sharpening. J Opt Soc Am 64:1200, 
1974), is an implementation of the first derivative of the image intensity. In 
spectral terms, this is a bandpass filter that enhances frequencies just below the 
5 highest in the image. Squaring the sum magnifies the differences between function 
values. 

F 2 is the ID Laplacian also described by Erteza, loc. cit, and Muller and 
Buffington, loc. cit. This filter is a measure of the second derivative of the image 
intensity. By operating on immediately adjacent pixels, F 2 has more predominant 
1 0 highpass frequency characteristics than F„ measuring resolution at a smaller scale. 
A variation of the Laplacian, based on lateral inhibition in the eye, was also 
evaluated by others, such as Harms and Aus (Harms H, Aus HM: Comparison of 
digital focus criteria for a TV microscope system. Cytometry 5:236-243, 1984). 
F3 is the sum of the squares of the difference filter as described by Erteza, loc. cit., 
15 and Muller and Buffington, loc. cit. By operating on both immediately adjacent 
pixels, F 3 has the most predominant highpass frequency characteristics and 
measures resolution at the smallest scale. The use of similar derivative filters is 
explored by others, such as Shazeer and Harris (Shazeer D, Harris M: Digital 
Autofocus Using Scene Content, in Architectures and Algorithms for Digital Image 
20 Processing II, SPIE 534:150-158, 1985). F 4 , a common 2D Laplacian not 
previously tested, was added for comparison. With square pixels, F 4 would have 
been a mixture of the highest frequencies, corresponding to the horizontally and 
vertically adjacent pixels, and the next highest frequencies, corresponding to the 
diagonally adjacent pixels. With the rectangular pixels and larger vertical 
25 sampling period of the RS-170 camera, however, this filter mixed in lower 
frequencies and did not have a higher frequency response than F 3 . 

B. Functions Based on Contrast 
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F 5 , the statistical variance of the intensity as a measure of contrast, was 
proposed by, for example, the Kernforschungszentrum Karlsruhe GmbH 
(Kernforschungszentram Karlsruhe GmbH: Verfahren und Einrichtung zur 
Automatischen Scharfeinstellung eines jeden Bildpunktes eines Bildes. Patent 
5 Specification PLA 7907 Karlsruhe, 1979). F 6 is the standard deviation of the 
intensity, or the square root of F 5 . It should be noted that under some conditions 
contrast achieves a local minimum, rather than a maximum, at best focus. The 
interference fringes that cause this are more commonly observed in transmission 
electron microscopy. With the light microscope, one way to observe this 
10 phenomenon is by using phase contrast to image a micrometer (e.g., 0.85 NA 40x 
objective and 10 /xm spacing). Best focus is at a local contrast minimum., and 
interference produces a series of contrast maxima and minima as focus is changed. 
Thus, contrast as a measure of focus must be utilized with caution in specimens 
with nonrandom spacing viewed in brightfield microscope modes. 

15 C. Functions Based on Combined Resolution and Contrast 

F 7 and F 8 combine the variance and standard deviation, respectively, and 
a 3 x 3 sharpening filter. As pointed out by others, such as Vollath (Vollath D: 
The Influence of the Scene Parameters and of Noise on the Behavior of Automatic 
Focusing Algorithms. J Microsc 152(2):133-146, 1988), the frequency spectrum 

20 is independent of the variance. That is, the variance can be changed by scaling the 
intensities without altering the relative Fourier power spectrum. The converse is 
not true: filtering the image can change the contrast. Thus, the image statistics 
measure a property fundamentally different from the Fourier spectrum, or 
sharpness of the image. This suggested using the variance (or standard deviation) 

25 as the basic autofocus measure and modifying the frequency effect by prefiltering 
the image. The fact that hardware capable of calculating the variance of a filtered 



WO 96/01438 PCT/US95/08424 



30 



image at video rates is becoming common makes consideration of this class of 
autofocus functions appropriate. 

D. Functions Based on Autocorrelation 

Correlation can be used to align images by multiplying them at different 
5 relative shifts and summing the resulting pixels. A maximum occurs when the 
images are correctly aligned. Similarly for autofocus, if an image is multiplied by 
itself shifted D i5 D p a maximum in the correlation function occurs at D ; = 0, Dj = 
0. Vollath, loc. cit, 1987 and 1982, pointed out that if the correlation function at 
Dj = 0, Dj = 0 is compared with the correlation function with the image shifted 
1 0 with respect to itself, say D, = 1, Dj = 0, the difference increases as image contrast 
increases. Therefore a maximum in the correlation function, F g , should occur at 
best focus. Vollath, loc. cit., 1988, then made analogies between F 8 and the 
variance, F 3 , to obtain F 9 and F 10 . These correlation functions apparently have not 
been previously tested on biologic microscope images. 

15 III. PERFORMANCE RESULTS 

To determine the suitability of the various functions using the sequential 
autofocus process of Figure 6, each was first tested on selected microscope fields 
for both phase contrast and fluorescence microscopy. In preliminary experiments 
it was noted that the shapes of the focus function plots were dependent on the 

20 number of cells in the field. In particular, it was found that a field containing a 
single small cell produced significantly different results than a field containing 
several cells. For this reason tests were performed on both types of fields. It was 
also found that the results were dependent on magnification, even using the same 
objective. Therefore, tests were performed at a series of magnifications by 

25 changing the zoom on the relay lens. From these experiments on selected fields, 
functions were chosen for phase contrast and fluorescence autofocus, compared in 
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experiments scanning many fields and evaluated for accuracy, precision, reliability 
and speed. 

A. Evaluation of Autofocus Functions on Selected Microscope Fields 
1. Microscope Field with Ten Cells 

The autofocus functions were first tested on a microscope field containing 
ten cells. The focus was changed in 0.098 /an increments (4 out of 4096 digital 
steps in a 100 /an range), and at each position the functions were evaluated for 
both phase contrast and fluorescence before moving to the next position. Figure 
7 shows the plots of scaled function value versus position for fluorescence. While 
there are clear variations in peak widths and sharpness, the functions are primarily 
unimodal. Only functions dependent on statistical measures (Figures 7e, 7f and 
7i) show side peaks, and those are probably not significant enough to cause an 
algorithm to focus incorrectly. These small side peaks also appear in the mean 
intensity (Figure 7e). The repeating pattern of these peaks indicates that they are 
probably due to interference, rather than lamp fluctuations. 

Table 2 summarizes the peak widths and best focus of each function for 
fluorescence. The widths at 90% of maximum show a clear dependence on the 
frequency characteristics of the function. Functions F h F, and F 3 are ordered from 
lowest to highest frequency enhancement and the peak widths narrow with higher 
frequency, giving F 3 the narrowest peak. F^ which looks similar to F, in spatial 
filter terms, also has very similar 50% and 90% widths. The resolution functions 
have very narrow peaks, whereas the contrast functions have much wider peaks. 
The combination functions, F 7 and F g , offer a trade off, with narrower peaks than 
the contrast functions and wider ranges than the resolution functions. The 
maxima, or best foci, for the predominantly statistical functions, F 5 , F 6 and F u 
differ by L07 pan from the others. Although one field is inadequate to determine 
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the significance of this difference, it raises the possibility that measures of contrast 
and resolution might not give the same focus. 

Table 2 

Autofocus Performance: Fluorescence, 10 Cells 



Function 


Widths (jaa) at 
Percent of Peak 
50% 90% 


Ratio 
50%/90% 


Best 
Focus 


Fl 


5.20 


1.85 


2.8 


48.742 


F2 


4.95 


1.70 


2.9 


48.742 


F3 


3.33 


1.13 


2.9 


48.742 


F4 


4.95 


1.65 


3.0 


48.742 


F5 


9.86 


2.96 


3.3 


47.668 


F6 


16.20 


4.20 


3.8 


47.668 


F7 


6.10 


2.45 


2.5 


48.742 


F8 


11.00 


3.35 


3.3 


48.742 


F9 


5.00 


1.70 


2.9 


48.742 


F10 


5.28 


2.00 


2.6 


48.742 


Fll 


9.90 


2.95 


3.4 


47.668 



The phase contrast results from the same experiment on a field with 
20 ten cells are shown in Figure 8 and Table 3. In Figure 8 it is immediately obvious 
that the peaks are not as sharp and that the plots are more irregular. There is a 
tendency toward side peaks in all the plots and these are especially prominent in 
Figures 8e and 8i with the statistical functions. It can also be seen that the 
tendency toward side peaks is reduced by higher frequency response filters, with 
25 progressive reduction in the first shoulder from F, through F 2 and F 3 in Figures 8a, 
8b, and 8c. The same trends as with fluorescence are visible in the peak widths. 
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That is, the highest frequency response filters, F 3 and F 4 , also have the narrowest 
peaks and the contrast functions have very wide ranges. 

Table 3 

Autofocus Performance: Phase Contrast, 10 Cells 
5 Widths (/an) at 





Percent of Peak 


Ratio 


Best 


Function 


50% 


90% 


50%/90% 


Focus 


Fl 


5.73 


1.95 


2.9 


49.621 


F2 


5.55 


1.92 


2.9 


49.621 


F3 


5.35 


1.88 


2.8 


49.621 


F4 


5.34 


1.80 


3.0 


50.012 


F5 


22.05 


2.55 


8.6 


49.915 


F6 


37.95 


3.35 


11.3 


49.915 


F7 


5.54 


1.90 


2.9 


50.012 


F8 


8.00 


2.20 


3.6 


50.012 


F9 


5.55 


1.95 


2.8 


49.621 


F10 


5.77 


2.09 


2.8 


49.621 


Fll 


26.25 


3.20 


8.2 


49.915 



2. Microscope Field with One Cell 

20 The next experiment was performed in the same way on a 

microscope field containing a single cell. From the fluorescence data shown in 
Figure 9 and Table 4, it can be seen that the peaks are narrower. This is probably 
caused by the reduced distribution of cellular components in the vertical direction. 
With more cells, it is more likely that portions will extend farther from the 

25 coverslip. This may be even more true with a nuclear stain as used here, since the 
nucleus usually causes a spherically shaped cellular bulge and is not directly 
adherent to the glass as is the cell membrane. When the depth of field is 
comparable to cell thickness, the width of the focus function will certainly depend 
on specimen thickness. Furthermore, functions F 2 and F 3 show 90% peak widths 

30 of only 0.12 /im in Table 4. This is considerably less than the theoretical depth 
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of field of 0.74 /xm. It may be because each result is a sum of a large number of 
pixels that is then squared. Summing a large number of pixels (245,760) increases 
the signal-to-noise ratio significantly. Squaring narrows the peaks further. The 
depth of field derivation by Born and Wolf, loc. cit, p. 441, assumes a 20% loss 
5 in vertical resolution at the extremes of the focal section as measured by 
attenuation of the central image patch. Evidently the signal-to-noise characteristics 
of this implementation allow significantly better discrimination than a 20% change. 

Table 4 



Autofocus Performance: Fluorescence, Single Cell 





Widths (urn) at 








Percent of Peak 


Ratio 


Best 


Function 


50% 


90% 


50%/90% 


Focus 


Fl 


2.95 


1.06 


2.8 


49.719 


F2 


2.10 


0.12 


17.5 


49.817 


F3 


1.65 


0.12 


13.7 


49.817 


F4 


4.28 


1.01 


4.2 


49.719 


F5 


13.68 


3.15 


4.3 


49.426 


F6 


17.85 


5.10 


3.5 


49.426 


F7 


5.81 


1.16 


5.0 


49.426 


F8 


12.78 


1.76 


7.3 


49.426 


F9 


2.13 


0.30 


7.1 


49.817 


F10 


3.45 


0.40 


8.6 


49.719 


Fll 


13.00 


3.55 


3.7 


49.231 



From Table 4, there were again differences between maYima j or best 
25 foci, with the largest between the statistical functions and highpass filters. These 
differences, however, are less than with the data from the field with ten cells, 
raising the possibility that specimen thickness may have played a role. 

The phase contrast data from the single cell experiment are shown 
in Figure 10 and Table 5. From Figure 10, it appears that phase contrast focus on 
30 a field with a single cell offered the most severe autofocus challenge. All the plots 
exhibit significant side peaks and some appear quite noisy ("noisy" here is 
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descriptive only; it is probably not true that image noise caused this appearance). 
The statistical functions F 5 , F 6 and F n in Figures lOe and lOi are both noisy and 
have the largest side peaks. F 4 in Figure lOd also appears noisy. At first glance 
it was tempting to attribute the noisy appearance of F 4 to the frequency 
characteristics of the highpass filter. However, as noted before, the frequency 
response of a camera with rectangular pixels lowers the vertical frequencies, 
complicating attempts to explain the cause. F 7 and F 8 in Figure lOf, mixtures of 
resolution and contrast functions, also appear noisy. The simple highpass filters 
F t , F, and F 3 in Figures 10a, 10b, and 10c are smooth and exhibit the earlier 
observed decrease in side peaks with increasingly high frequency response. From 
Table 5, there are again some differences in best foci, but these differences are 
small. 

Table 5 



Autofocus Performance: Phase Contrast, Single Cell 





Widths (jim) at 








Percent of Peak 


Ratio 


Best 


Function 


50% 


90% 


50%/90% 


Focus 


Fl 


4.85 


1.42 


3.4 


51.282 


F2 


4.12 


1.23 


3.3 


51.282 


F3 


3.40 


1.03 


3.3 


51.282 










51.477 


F5 


8.40 


1.88 


4.5 


51.477 


F6 


8.62 


2.00 


4.3 


51.477 


VP 








51.477 


F8 f 








51.477 


F9 


4.53 


1.21 


3.7 


51.282 


F10 


5.30 


0.62 


8.5 


51.282 


Fll 


11.82 


2.30 


5.2 


51.477 



* Widths could not be determined due to multimodality (see Fig. 4) 
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Some indications of the sensitivity of the functions to lamp 
fluctuations in phase contrast can be seen in Figures 8 and 10. In both of the 
corresponding experiments, there were intensity spikes. In Figure 8e, there is a 
mean intensity spike at the position of about 87 fim, and in Figure lOe, one at near 
5 55 /an and another at 85 /on. The mean intensity spike in Figure 8e showed up 
in F 4 , F 7 and F 8 in Figures 8d and 8f and slightly in F 5 , F 6 and F, , in Figures 8e 
and 8i. The resolution functions F, - F 3 in Figures 8a-8c and the autocorrelation 
functions F 9 and F 10 in Figures 8g and 8h appear to have been immune from this 
lamp fluctuation. This same pattern is exhibited in Figure 10. It is interesting to 
10 note that, with the exception of F 5 , the functions that were sensitive to these lamp 
fluctuations are dependent on the contrast measures of variance or standard 
deviation. 

3. Function Dependence on Magnification and Sampling The data 
for phase contrast focus on a single cell suggested that the frequency response of 

1 5 the focus function plays an important role in the formation of side peaks. It is 
likely that these side peaks arise from interference just above and just below best 
focus. Interference would be expected occur at lower frequencies since the 
departure from best focus degrades the modulation transfer function (MTF) of the 
microscope creating a lower frequency cut off. If a focus function measured only 

20 the highest frequencies it should be immune from these effects. The focus 
function, however, is only one source of the frequency response. The microscope 
and the camera also have characteristic frequency responses that act prior to the 
focus function. Ideally, the camera should sample with at least twice the 
maximum frequency of the optical signal, according to the Nyquist sampling 

25 criterion. The Rayleigh resolution estimate, has been explained by others, such as 
Inoue (Inoue S, Video Microscopy. Plenum Press, New York, 1986) is 
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rf _ L22A 

where X is the wavelength of light and NA is the numerical aperture. With a 0.52 
NA condenser, a 0.75 NA objective, and a peak wavelength of 540 nm 
corresponding to the peak transmittance of the daylight filter utilized, the 
resolution was 0.518 /un. Thus, the image should have been magnified so that the 
5 distance between components of the specimen projected onto adjacent pixels 
corresponded to about 0.25 /an in the specimen. At a zoom of 1.0 with these 
optics, the projected distance was 0.607 fun. This represents a condition of 
undersampling and causes aliasing. Achieving Nyquist sampling would have 
required a zoom of 0.607/0.250 = 2.43, above the maximum available zoom of 
10 2.25. 

Unfortunately, the limited brightness in fluorescence microscopy can 
make Nyquist sampling highly impractical and even impossible. Intensity is 
proportional to the inverse square of the magnification and even with the bright 
preparation used here, a zoom of 2.25 forces operation of the camera and image 
processor at the upper limits of gain, resulting in a very noisy signal. Limited 
fluorescence intensity motivates the use of high NA, low magnification objectives 
that increase the problem of undersampling. Because of the signal loss with 
increased magnification, it is impractical to optimally sample for autofocus in 
fluorescence and undersampling conditions were maintained for these experiments. 

It is important, however, to understand the effects of sampling on 
autofocus, since many objectives and microscopes with different magnifications 
and NA's are available. To further study the dependence of the side peaks on 
magnification, an experiment on a microscope field with a single cell was carried 
out at a series of zooms, 0.9 - 2.25, that correspond to a range of 37 - 93% 
undersampling. A 3D plot of the response of function F 3 versus focus position and 
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zoom is shown in Figure 11. At a zoom of 0.9 the side peaks are big enough to 
cause an autofocus algorithm to locate a spurious best focus under some 
conditions. As predicted, increasing the magnification to nearly optimal sampling 
at 2.25 caused the side peaks to disappear. This experiment underscores the fact 
5 that the choice of optics and camera can be very important in determining focus 
function characteristics: anything that changes the resolution or the magnification 
will cause similar changes. 

It was also observed, for example, that growing the cells on the 
coverslip instead of the slide increased the size of the side peaks (data not shown). 

1 0 This was due to the gain in resolution from placing the cells at the location where 
objective aberrations are best corrected as described by others, such as Gibson 
(Gibson SFF, Modeling the 3-D Imaging Properties of the Fluorescence Light 
Microscope, Ph. D. Dissertation, Carnegie Mellon University, Pittsburgh, 
Pennsylvania, 1990). The index of refraction of the mounting media, thickness of 

15 the coverslip, dirt in the optical path, illumination wavelength, and camera and 
image processor electronics are other components that can alter the system MTF, 
such as described by Inoue, and change the shape of the focus function. 

B. Autofocus Performance in Automated Scanning 

Finally, phase contrast and fluorescence autofocus were tested in a 

20 series of experiments scanning rectangular areas of > 1 000 fields. The purpose of 
these experiments was to test the hypothesis that the weighted average is a good 
estimate of best focus, measure autofocus precision, determine how small a number 
of focus positions could be used without compromising precision in an attempt to 
achieve maximum speed, and compare the best focus between phase contrast and 

25 fluorescence. 

Several rectangular areas on different specimens were scanned in a 
raster pattern, with refocusing performed 20 times in fluorescence and then 20 
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times in phase contrast F 3 was used in phase contrast for all experiments and 
either F 3 or a variation of F 7 , where the filter was [-1 2.5 -1], were used for 
fluorescence. These choices were based on consideration of peak sharpness and 
unimodality. With scanning microscopy, a operation over a large vertical range 
5 is not as important because the best foci of adjacent fields are usually not far apart 
In addition, those functions resulting in the largest range also had problems with 
unimodality on a single cell in phase contrast. Therefore, the highest frequency 
response filter was chosen for phase contrast. For fluorescence, F 3 gave a narrow 
enough range with a single cell (1.03 jun 50% peak width from Table 5) to be a 
10 problem even for scanning microscopy. Therefore, F 7 and F 3 were considered 
good candidates. Since the real-time implementation utilized the interlaced camera 
signal, the variation of F 7 substituting a ID sharpening filter for the 2D filter was 
used. 

1. Accuracy, Precision and Speed 

1 5 F° r each set of 20 autofocus tests, mean and standard deviation were 

calculated for both the maximum and the weighted average of best focus. The 
differences between the means of the maxima and weighted averages were also 
calculated to determine if the weighted average was a comparatively accurate 
estimate of best focus. 

20 The results of these tests are shown in Figure 12. From the 

combined standard deviations, the autofocus precision in phase contrast averaged 
0.154 fsm with the maximum and 0.069 jtm with the weighted average. In 
fluorescence the precision averaged 0.230 fim for the maximum and 0.159 for 
the weighted average. This is considerably better than the 0.74 /an depth of field 

25 of the 20x, 0.75 NA objective. In all but the first experiment the precision was 
better with phase contrast than with fluorescence. There are a number of factors 
that could have contributed to this difference. In phase contrast, the image was 
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strobed near the end of the video field after the piezoelectric focus had stopped at 
its new position, whereas in fluorescence, each field was integrating on the CCD 
while focus was changing (30 - 50% of the field duration). Also, as previously 
discussed, the cellular and nuclear components may have been distributed 
5 differently. 

The statistics from the difference of the maxima and weighted 
averages showed a very good agreement between the two estimates of best focus. 
In phase contrast, the differences ranged from -0.025 to 0.014 jim and the largest 
standard deviation was 0.071 /im. In fluorescence, the differences ranged from - 

10 0.002 to 0.044 with a maximum standard deviation of 0.088 jan. Given this 
agreement between the two estimates and the improvement in combined standard 
deviation, it is clear that the weighted average was a better measure of best focus. 

The above performance was obtained with focus times as short as 
0.25 second (s) in phase contrast. There appeared to be no degradation in focus 

1 5 precision at 0.25 s with 1 1 focus positions tested. Therefore, even faster autofocus 
may be possible. 



2. Phase Contrast Focus as an Estimate of Fluorescence Focus 

The differences between the means of each set of focus tests in phase 
and fluorescence are shown in Table 6. Excluding experiment 2, where the 

20 fluorescence autofocus lost track for part of the scan, the average of the differences 
between the two microscope modes varied between -0.394 and 0.745 /im, with 
good agreement between the maxima and weighted averages. There are many 
possible causes of this difference in foci, including microscope alignment, focus 
sampling interval and differences between nuclear and cytoplasmic component 

25 distributions. These results indicate that measuring and correcting for the 
difference between the two microscope modes may yield significant improvement. 
It did not seem possible, however, to predict the difference from one specimen to 
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another. Again excluding experiment 2, the average standard deviation was 0.347 
jim for the maximum and 0.342 for the weighted average. Thus the standard 
deviation of the difference was about 1/2 the depth of field of the objective. 
Although this might be enough to cause a small loss of precision in fluorescent 
5 measurements, it indicates that phase contrast autofocus provided a good estimate 
of fluorescence autofocus. 

Table 6 

Comparison of Best Focus between 
Phase and Fluorescence in Automated Scanning 

0 Mean Phase - Mean Fluorescence 



Maximum Weighted Averages 





Experiment 


Mean 




Mean 






1 


0.292 


0.420 


0.293 


0.410 


15 


2* 


0.934 


1.150 


0.985 


1.140 




3 


0.719 


0.235 


0.745 


0.219 




4 


-0.183 


0.142 


-0.196 


0.137 




5 


-0.394 


0.457 


-0.349 


0.465 




6 


-0.214 


0.397 


-0.175 


0.400 


20 


7 


-0.301 


0.433 


-0.304 


0.423 



All measurements in microns. 

* Fluorescence autofocus lost track for a portion of the scan. 



IV. CONCLUSIONS 
The experiments carried out with the presently preferred embodiment of the 
25 autofocus system of the present invention showed that it is possible to scan large 
areas of a microscope slide using phase contrast autofocus to minimize exposure 
for fluorescence imaging. This should make possible the imaging of living cells 
with minimal toxicity and the analysis of sensitive fluorescence preparations 
without photobleaching while focusing. There was a significant difference 
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between best focus in phase contrast and fluorescence, but the difference was 
constant enough to allow correction. In addition, it was shown that autofocus can 
be performed with precision an order of magnitude better than the depth of field 
in less than 0.25 s. Improved precision was achieved using the weighted average, 
5 which made use of the data from all focus positions tested. 

The power-weighted average is the average of the positions, each weighted 
by a power of the focus value at that position. The weighted average is calculated 
from the array of focus function results. Each focus function result is a magnitude 
resulting from calculation of the function over the entire image frame (245,760 

10 pixels) or field (122,880 pixels) at a single position. The mavimnm does not 
account for the magnitude of the values at adjacent positions, while the weighted 
average does by averaging the function values at all positions. 

This performance was achieved after minimizing the effects of 
undersampling by choosing autofocus functions with the most prominent highpass 

15 filter characteristics. The problem of undersampling could be even more severe 
with lower magnification, high NA objectives and higher NA condensers. 
Multimodality was even more severe for functions less dependent on resolution 
and particularly severe with contrast measures. The lack of unimodality with 
intensity variance (or standard deviation) in phase contrast autofocus makes use of 

20 contrast-based functions more questionable. The interference postulated to cause 
this problem may be present with all forms of transmitted microscopy where the 
image elements are small in number or regularly spaced. 

For fluorescence, the fact that brightness is also directly dependent on 
distance from best focus may overwhelm the unwanted interference contrast 

25 extrema. All autofocus functions tested here decrease in magnitude with 
decreasing image intensity. The attenuated fluorescence may decrease or eliminate 
multimodality in the contrast measures of focus. The highpass filter functions 
have an even narrower range because of this intensity dependence. By combining 
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the statistical and resolution measures, the range can be broadened while retaining 
a relatively narrow peak. Such a combination may be important for scanning 
sparsely populated cell specimens and necessary for autofocus applications 
requiring greater operating range. 

The level of autofocus reliability and speed achieved here is an important 
step in bringing measurements common in flow cytometry closer to practical use 
in scanning cytometry. Such measurements may have advantages related to in situ 
analyses, such as morphology, relationship and position not possible with flow 
cytometry. Position may be a particular advantage for time lapse analysis of living 
cells where cell-by-cell tracking would be possible with short scan intervals. 

AUTOFOCUS USIN G VOLUME IMAGING WITH CONTINUOUS SCANNING 
The speed of my autofocus technique may be increased with a volume 
imaging apparatus that continuously acquires an image of the volume of an image 
object that is being examined by a microscope. The image includes a set of 
15 images obtained at respective two-dimensional image planes that are offset 
vertically. Hereinafter such an image is referred to as a "volume image" or a 
"three-dimensional image". 

This embodiment is based upon my critical realization that a volume image 
of a portion of an image object being examined by microscope can be obtained by 
20 an apparatus that includes (1) a plurality of imaging devices, each imaging device 
positioned at a respective image plane of a plurality of substantially parallel image 
planes, where each image plane is displaced in a particular dimension with respect 
to each other image plane, and (2) an image forming apparatus coupled to the 
plurality of imaging devices that forms a plurality of electronic image 
25 representations, each electronic image representation representing an image at a 
respective image plane of the plurality of image planes. 



5 



10 
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Each electronic image representation represents a section of the image 
object that is displaced in the particular dimension from all other sections of the 
same image object. 

To support autofocus in a scanning microscope, the volume imaging 
5 apparatus is scanned continuously ahead of a specimen imaging apparatus. Both 
the volume imaging apparatus and the specimen imaging apparatus observe the 
same image object through the same microscope optics. Therefore, autofocus can 
be implemented by the analytical methods of my autofocus technique described 
above with respect to Figures 1-12. As will become clear, the volume imaging 

10 apparatus does not require incremental positioning of the microscope in the Z 
dimension in order to obtain a volume image. Therefore, it is not necessary to 
stop at each microscope field to obtain a set of images by incrementally 
positioning the microscope stage in the Z dimension. Accordingly, this 
embodiment achieves a very important objective of speeding up my autofocusing 

15 technique illustrated in Figures 1-12. 

Figure 13 illustrates, in hybrid block/schematic form, the invention in a 
system that fully automates continuously-scanning autofocusing in a microscope 
390. The microscope 390 comprises two overlapping optical systems, one for 
volume imaging, and one for fluorescence imaging. Preferably, the microscope 

20 390 is an epifluorescent upright microscope (such as that described above with 
reference to Figure 1) with phase contrast optics. The fluorescence imaging 
optical system includes an epi-illumination light source 410 which illuminates an 
image object, with the aid of a collector lens 412 and a relay lens 414, a 
fluorescence filter block 415, with its emission interference filter removed and 

25 placed at 448, and a piezoelectric-positioned objective lens 418 that is adjustable 
bidirectionally along the arrow 419. The epi-illumination provided by the light 
source causes an image object (in this case, a microscope field 420 with cells) to 
fluoresce, with the fluorescent emissions passing through the objective lens 419 
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and the filter block 415 to a dichroic mirror 430 which reflects the fluorescent 
emissions through the interference filter 448 and a relay lens 432 onto a CCD 
camera 433. The output of the camera 433 is provided to a cytometry processor 
434 for processing fluorescent images of the cells in the microscope field 420. 

The volume-imaging optical system includes a trans-illumination light 
source 400 that provides illumination of the image object for phase-contrast optics 
including a collector lens 402 and a relay lens 404, a mirror 406 which reflects the 
trans-illumination through a wide band pass interference filter 407 (for example, 
a 600 nm ± 50 nm bandpass filter) which separates light used for autofocusing 
from the fluorescence excitation and emission wavelengths (typically, 365nm and 
480nm for DAPI). Manifestly, the trans-illumination wavelength can easily be 
changed to accommodate other fluorescence wavelengths. The trans-illumination 
light is separated from the emitted fluorescence light by the dichroic mirror 430 
and imaged through the relay lens 435 with a volume imaging apparatus that 
includes a volume i m aging array 436 and an array of cameras 437. The cameras 
of the array 437 can comprise, for example, time-delay-and-integration (TDI) CCD 
cameras. Preferably, the cameras of the array 437 operate in parallel, with their 
outputs (hereinafter, "electronic image representations") converted from analog to 
digital (A/D) format through A/D circuits 440. The parallel converted electronic 
image representations are provided from the A/D circuit 440 to a volume image 
processor 442 (that includes storage 443) for calculating a best focus from the 
electronic image representations of the camera array 437. The best focus output 
of the volume image processor 442 is provided to a piezoelectric objective 
positioner controller 446 that operates a piezoelectric objective positioner 447. 
The piezoelectric objective positioner 447 positions (or repositions) the objective 
lens 418 at (to) the current best focus position as determined by the volume image 
processor 442. 
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The microscope 390 is motorized by means not shown for scanning in the 
xy plane and for movement on the Z axis as indicated by the axes 439. In this 
respect, the microscope 390 is identical with the microscope 102 of Figure 1. 

The optical principle of volume imaging according to the invention may be 
understood with reference to Figure 14A and Figure 14B. Volume imaging is 
based on the idea that within a reasonable vertical distance (several microns) of 
best focus at an image forming microscope plane ("image object plane") is an in- 
focus volume consisting of adjacent optical planes. This can be demonstrated by 
a simple experiment performed on a microscope with a video camera and display. 
First, an image object is placed in focus and displayed on the display. Then the 
video camera is moved a short distance along the optical axis. This will cause the 
image to defocus. Next, with the new position of the video camera retained, the 
focus knob on the microscope is adjusted to reacquire sharp focus. This image 
object-objective focal distance is changed by a few microns in this last step. Thus, 
relative to the objective face, a new optical plane is brought into focus. Within 
a reasonable vertical distance, this does not degrade optical performance. The 
means for parallel acquisition of such an image volume is shown in Figures 14A 
and 14B. 

In Figure 14A, there is shown an array 470 of optical light pipes (optical 
fibers). The optical light pipes of the array 470 are coupled to an array 474 of 
CCD cameras. In this regard, each optical light pipe has two ends, a first end 
being held together with corresponding first ends of the other optical light pipes 
in a stepped array that is magnified in Figure 14B, and a second end connected to 
a respective CCD camera. For example, the optical light pipe 472 includes first 
and second ends, 473 and 474, with the second end 474 coupled to the CCD 
camera 475. The optical light pipe 476 includes first and second ends 477 and 
478, with the second end 478 connected to the CCD camera 479. The first ends 
473 and 477 of the optical light pipes 472 and 476 are held together in the stepped 
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array illustrated in Figure 14B. As Figure 14B shows, the first ends of the optical 
light pipes, including the ends 473 and 477 are disposed in substantially 
longitudinal array, with each respective first end being positioned at a respective 
image plane of a plurality of image planes. For example, the end 473 is 
5 positioned at an image plane designated as Z„ while the end 477 is positioned at 
a second image plane denoted as Z^ Figure 14B shows that, for n light pipes, the 
stepped, substantially longitudinal array of optical light pipe first ends defines a 
corresponding array of image planes where each image plane is displaced in a 
particular direction (the Z dimension of this embodiment) with respect to each 

1 0 other image plane. Thus, the image plane Z, is vertically upwardly displaced from 
the image plane Z^ while the image plane Z n is vertically downwardly displaced 
from the image planes Z, and Z^ The stepped array of first ends is offset in the 
Z direction from an image object plane (a surface of a microscope slide, for 
example) on which an image object (a stained cell, for example) rests. 

1 5 Manifestly, each optical light pipe constitutes an imaging device and each 

camera is an image forming apparatus. Referring again to Figure 13, the array of 
optical light pipes 470 corresponds to the volume imaging array 436 of Figure 13, 
while the array 474 of CCD cameras corresponds to the camera array 437 of 
Figure 13. 

20 Figures 15A and 15B illustrate a second embodiment of a plurality of 

i m aging devices according to the invention in the form of a linear image sensor 
481 having a linear array 482 of active photo elements, such as the photo element 
485. The image sensor 480 is tilted with respect to an image object plane 488 
where an image object is located so that each photo element intersects a respective 

25 image plane which is displaced in the vertical dimension with respect to each other 
image plane. Thus, in Figure 5B, the image plane Z i is intersected by a photo 
electric element 485, while the image plane Z M , offset by vertical distance d, is 
intersected by another photo electric element 486. With this embodiment of 



WO 96/01438 PCT/US95/D8424 



48 



imaging devices, the images are formed by CCD camera electronics which operate 
in the same manner as the electronics of the cameras in the array 474 of Figure 13. 

With reference now to Figure 13, Figure 14A and Figure 14B, the 
microscope 370 includes a conventional stage upon which a microscope slide 438 
5 is carried. The microscope slide 438 is prepared for fluorescent cytometry as 
described above and is scanned over a plurality of microscope fields in a raster 
pattern automatically as described above. Such scanning is in the xy plane of the 
axes 439. With the stage moving at a constant velocity in x, adjacent image 
planes are sampled by the cameras of the CCD camera array 474 with small, 

10 successive time delays. These delays are removed by buffering. The fluorescence 
image acquired for analysis is collected with a longer delay as shown in Figure 5 
to allow for the piezoelectric repositioning time, for example, a maximum of 10 
msec, or < 100 micrometer travel in a 500 micrometer square field with a 20x 
objective. Focus can be calibrated by measuring the difference between the 

15 fluorescence image acquired for analysis and the volume image acquired for 
focusing. In this regard, the differences between the best fluorescence and volume 
imaging focuses may be measured by repeated scans at slightly different focus 
levels to compute focus directly from the fluorescence images. The fluorescence 
focus position may then be set to eliminate the difference between the best focus 

20 in fluorescence and volume imaging for subsequent scans. 

As shown in Figure 16, the stepped ends of the optical light pipe array 470 
form a volume imager field of view (FOV) that scans in the scan direction over 
a portion of a microscope field such as the field 490. Trailing behind the volume 
imager FOV is a fluorescent imager FOV. With respect to the volume imager 

25 FOV, consider a slice of the field having an aspect ratio approximately 
corresponding to that of each of the rectangular first ends in the stepped array of 
optical light pipe ends including the ends 473 and 477. The slice is indicated by 
A, in Figure 16. As the image planes Z,-Z n are scanned over the slice, a two- 
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dimensional image of the slice is captured by each optical light pipe/CCD camera 
combination. The image can comprise one or more pixels. Each image, denoted 
as P, is stored (as an electronic image representation) in a buffer corresponding to 
the slice in the manner shown in Figure 17. Thus, for a set of successive buffer 
5 locations in the buffer for the slice Aj ( ,l the Aj buffer"), as each first end face of 
the stepped end face array shown in Figure 14B is scanned over the slice, an 
electronic image representation representing a two-dimensional image at the image 
plane of an end face is stored at a storage location corresponding to the end face. 
Thus, an image P, formed at image plane Z, corresponding to the end face 473 of 

10 the optical light pipe 472 is stored in the A ; buffer at the position corresponding 
to image plane Z,. In this manner, as the stepped end faces of the optical light 
pipe array 470 are scanned over the slice, a plurality of electronic image 
representations Pj are stored in the 4 buffer. As Figure 17 shows, there are 
buffers provided for an array of slices. Figure 18 shows a volume image stored 

15 in the Aj buffer formed by electronic image representations of the two-dimensional 
images P,-P n obtained through the image planes Zj-Z*, respectively. 

It may be appreciated that an advantage is gained when the volume image 
is acquired by "tilting" at the image plane (by offsetting the image planes in the 
vertical direction) rather than by directly tilting the microscope slide. The 

20 advantage is due to the feet that vertical magnification is the square of lateral 
magnification. In the embodiment illustrated in Figures 1-12, for example, 
magnification to the video camera CCD chip is about 3 Ox. Therefore, the vertical 
magnification is about 900x and a 4 pan thick volume of a specimen would be 
magnified to 3.6 mm at the image plane. This exaggerated vertical magnification 

25 simplifies precise sampling of the required optical sections. 

I have calculated dimensions of the step array of optical light pipe ends 
illustrated in Figure 14B based upon assumptions about the dimensions of image 
objects and available CCD camera technology. The dimensions of this array 
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(shown in Figure 14B) were calculated assuming a 4 jim thick specimen volume 
magnified through a 20x objective and a 0.9x-2.25x zoom relay lens (available 
from Nikon) onto 1024 pixel x 96 line TDI CCD arrays with 13 x 13 ^m pixels. 
However, design considerations, particular requirements of an application, and 
5 changes in optical and camera technology will manifestly vary these assumptions. 
Nevertheless, assuming relatively rigid optical light pipes, a well-supported, single 
package that integrates the optical light pipes with the CCD cameras is required 
for the embodiment illustrated in Figure 14A and 14B. The package is not shown, 
but may consist, for example, of a rectangular aluminum frame attaching to both 
10 the CCD cameras and the optical light pipes. Assuming a relatively small remote 
CCD camera (such as a 2.0" x 2.0" x 0.75" camera available from Dalsa) these 
components may be placed in a conveniently sized package. 

The dimensions shown in Figure 14B include a vertical step of 0.017" 
between optical light pipe end faces for convenience in assembly. This 
corresponds to a 29.387x lateral magnification and an 863.6x vertical 
magnification. Using the zoom lens described above, this magnification 
corresponds to a zoom of 1.04x (out of a possible range of 0.9x-2.25x). A 20x 
phase/fluorite objective having a field of view with a 710 ^m diameter for a 502 
Mm inscribed square field may be used. Such a field therefore would be magnified 
to 14.75 mm (0.58") square at the proposed lateral magnification. This leaves 
ample room for a trailing CCD that acquires the fluorescence image. The CCD 
cameras are disclosed as being of the TDI type to increase light sensitivity, which 
permits the dwell time to be reduced for continuous scanning. 

Manifestly, there will be light loss through the optical light pipe interfeces. 
However, this can be compensated adequately by the brightness range of 
conventional transmitted microscope lamps. Use of TDI cameras may also 
decrease a small percentage of random rearrangements of optical fibers common 
in optical light pipe interfaces and problems with Moire patterns that can occur 
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with improper matching of such optical fibers and at the camera arrays. As is 
known, any Moire set will be averaged out by TDI cameras. The 2:1 binning of 
the TDI cameras, both for fluorescence and phase contrast, may also eliminate 
undersampling and aliasing, without the loss of light accompanied by the usual 
method of limiting the effective numerical aperture (NA). 

Figure 19 illustrates the processing that is used to determine an optical 
quality metric by processing the image representations according to an autofocus 
function described in the incorporated patent application. Preferably, the function 
is a resolution function (described in Table 1) which sums the squares of the result 
of highpass filter (HPF). Relatedly, the squared values of the electronic image 
representations provide augmented intensity values. As Figure 19 shows, an 
electronic image representation representing an image at a respective image plane 
is acquired at the output of a CCD camera at step 501, digitized at step 503, and 
buffered at step 504. The sequence 501, 503, 504 continues until an electronic 
image representation for each image plane of a plurality of image planes is 
acquired and buffered. A measure of resolution is acquired by summing the 
squares of the result of highpass filtration of the digitized images in steps 505 and 
506. A focus metric is constructed in step 507 and the image plane where the best 
focus occurs as indicated by the focus metric is chosen in step 508 and the 
microscope is refocused by the autofocus mechanism, if necessary. A 
representative focus metric is illustrated in Figure 20 as a plot of a sharpness index 
indicating the measure of resolution at each image plane. 

Calculation of the optical quality metric (resolution, or sharpness) requires 
processing of the image representations. Assuming nine image planes such as 
would be acquired by the stepped end face of Figure 14B, nine parallel electronic 
image representations would have to be processed. In combination, this would 
produce a 90 MHz pixel rate that must be processed in parallel. Preferably, for 
each electronic image representation, a 1 x 3 convolution (-12-1) will be 
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applied for high pass filtration, followed by calculation of the sum of the squares. 
Manifestly, the volume image processor (442 of Figure 13) should be capable of 
processing the image representations in parallel. Such parallel processing is well- 
supported by pipeline image processors such as the Imaging Technology MVC 40 
5 MHz image processor. Such a system comprises a set of mother boards, each 
having a three-channel, eight-bit A/D converter, image buffers, and processing 
pipelines. For example, nine image streams can be processed in an Imaging 
Technology processor by three sets of mother boards. 

The volume image processing of the invention can be further improved by 

1 0 relocating the highpass filtration function. For example, in the embodiment of the 
autofocus system illustrated in Figure 21, the highpass filtration function is located 
in the optics of the microscope 390 at a backfocal, Fourier or pupil plane. In this 
embodiment, the plurality of imaging devices comprise an array 610 of photo 
elements that corresponds to the tilted linear array of high sensitivity photo electric 

15 elements shown in Figure 15A and Figure 15B. In this case, the array 610 of 
photo electric elements is coupled to camera electronics 620. In the embodiment 
of the autofocus system illustrated in Figure 21 (in which reference numerals 
identical with those in Figure 13 represent identical elements), the volume image 
processor 442 processes highpass filtered electronic image representations to 

20 produce the optical quality metric; this processor is simplified in that the electronic 
image representation output by the camera electronics 620 has the form of the 
integral 1^ shown in Figure 21. Manifestly, this integral reduces the cost of 
performing autofocus calculations, when compared with the volume imaging 
embodiment illustrated in Figure 13. In Figure 13, the volume image processor 442 

25 must process many two dimensional image representations that make up the three 
dimensional image. With a highpass function is performed optically as shown in 
Figure 21, the electronic image representation processing is reduced. With the use 
of Fourier relay optics 602 to produce a projected pupil plane (which is a Fourier 
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transform plane), an optical highpass filter 605 can be implemented as an 
attenuation (amplitude), or phase filter. See "Optical Information Processing," 
(S.H. Lee, Springer-Verlag: New York, 1981) for the principles of optical 
filtration. In operation, the optical highpass filter 605 blocks the low frequency 
5 portion of an image. The high frequency, or edge information is retained, and the 
intensity of the high frequency information is proportional to focus. Thus, the 
intensity at a particular image plane provides a direct measure of sharpness and the 
focus metric of Figure 20 can be derived by simply plotting intensity (as 
sharpness) versus image plane. The intensity (sharpness) at many different image 
10 planes is measured using the tilted high sensitivity photo electric element array 
610. This requires only a single image processor to buffer and remove delays 
from the electronic image representations and to calculate best focus. Essentially, 
the autofocus system embodiment of Figure 21 reduces the cost of implementation 
by replacing three image processing boards, nine CCD cameras and one optical 
1 5 light pipe array with one image processing board, one tilted photo electric element 
array, and one set of camera electronics. 

Continuous volume imaging could also rapidly scan a volume using the 
confocal microscope principle, as with either slit or spot confocal apertures. With 
a slit aperture, longitudinal CCD arrays on the CCD cameras which form images 
20 at image planes may be illuminated through a slit confocal aperture. Assuming 
linear CCD arrays in each camera of a single pixel in width, each linear CCD 
array would image a single plane. However, the image object would be 
illuminated with a corresponding confocal slit of light for each linear CCD array, 
resulting in higher resolution, especially in the vertical direction, but also laterally. 
25 Thus, an array of stationary slits, each illuminating a respective image plane, in 
combination with moving a microscope stage at a constant velocity in one 
direction, will create a real-time volume-imaging confocal microscope. As with 
the array of confocal slits, an array of confocal spots such as is taught in U.S. 
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Patent 5,239,178 could also be arranged at different image plane positions and 
combined with stage scanning to create even higher resolution. 

Clearly, other embodiments and modifications of the present invention will 
occur readily to those of ordinary skill in the art in view of these teachings. 
5 Therefore, this invention is to be limited only by the following claims, which 
include all such embodiments and modifications. 
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CLAIMS 

1 . A method of autofocus for a microscope having a focus mechanism, 
the method comprising the steps of: 

(1) acquiring a digital image of a specimen on a microscope stage 
5 at a first vertical position; 

(2) applying a digital filter to the digital image to obtain a filtered 

image; 

(3) applying a measurement function to the filtered image to obtain 
a magnitude; 

10 (4) moving the focus mechanism in the vertical direction to a 

second vertical position and repeating steps (l)-(3); and 

(5) comparing the magnitudes of the filtered images acquired at the 
first and second vertical positions so as to adjust focus of the microscope 
based on the comparison. 

15 2. The method defined in Claim 1 , wherein the digital filter performs 

an image sharpening transformation. 

3. The method defined in Claim 1, wherein the measurement function 
measures contrast 

4. The method defined in Claim 1 , wherein the measurement function 
20 includes the step of calculating the sum of the filtered image intensities. 

5. The method defined in Claim 1 , wherein the measurement function 
includes the step of calculating a sum of the squared filtered image intensities. 
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6. The method defined in Claim 1, wherein the comparing step 
includes the step of moving the focus mechanism toward the first vertical position 
if the magnitude at the first position is greater than the magnitude at the second 
position, and moving the focus mechanism toward the second vertical position if 

5 the magnitude at the first position is less than the magnitude at the second position, 
thereby adjusting the focus of the microscope. 

7. An apparatus for generating a volume image, comprising: 

a plurality of imaging devices, in which at least one imaging device 
is positioned at a respective image plane of a plurality of image planes, 
10 where each image plane is displaced in a third dimension with respect to 

each other image plane; and 

an image forming apparatus coupled to the plurality of imaging 
devices for forming a plurality of electronic image representations 
representing images at respective image planes of the plurality of image 
1 5 planes. 

8. The apparatus of Claim 7, further including processing means 
coupled to the image forming apparatus for determining an optical quality metric 
by combining electronic image representations. 

9. The apparatus of Claim 7, wherein the plurality of imaging devices 
20 includes a plurality of optical conductors, each optical conductor having a first end 

face positioned at a respective image plane and a second end face coupled to the 
image forming apparatus. 

10. The apparatus of claim 7, wherein the plurality of imaging devices 
are disposed in a substantially elongate array. 
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11. The apparatus of Claim 7, wherein the image forming apparatus 
includes a plurality of charge-coiqrfed device (CCD) cameras. 

1 2. The apparatus of Claim 1 1 , wherein the plurality of imaging devices 
includes a plurality of optical conductors, each optical conductor having a first end 

5 face positioned at a respective image plane of the plurality of image planes and a 
second end face coupled to a respective CCD camera. 

13. The apparatus of Claim 12, wherein each first end face is offset in 
the third dimension from each other second end face. 

14. The apparatus of Claim 7, wherein the plurality of imaging devices 
10 includes an elongate array of photo electric devices. 

15. The apparatus of Claim 14, wherein the elongate array of photo 
electric devices is tilted with respect to an image object plane, each image plane 
of the plurality of image planes being intersected by at least one photo electric 
device of the elongate array of photo electric devices. 

15 16. The apparatus of Claim 8, wherein the processing means includes: 

a plurality of image buffers, each image buffer for storing a 
plurality of electronic representations of at least a portion of an image; 

a high-pass filter coupled to a respective buffer for providing a 
filtered electronic representation; and 
20 means coupled to the high-pass filter for combining a plurality of 

filtered electronic representations to produce the optical quality metric. 
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17. The apparatus of Claim 16, wherein the means for combining 
includes: 

means for obtaining augmented intensity values for the filtered 
electronic representations; and 
5 means for indicating a value of the optical quality metric for each 

image based upon the augmented intensity values. 

18. The apparatus of claim 7, further including: 

an optical high pass filter positioned between the plurality of 
imaging devices and an image object plane; and 
10 processing means coupled to the image forming apparatus for 

determining an optical quality metric by combining electronic range image 
representations. 

19. The apparatus of claim 1 8, wherein the processing means includes: 
a plurality of image buffers, each image buffer for storing 

15 a plurality of filtered electronic image representations of at least a 

portion of an image; and 

means coupled to the plurality of image buffers for 
combining a plurality of filtered electronic image representations to 
produce the optical quality metric. 

20 20. The apparatus of claim 19, wherein the optical quality metric is 

image sharpness. 

21 . A method for volume imaging using a plurality of imaging devices 
in which at least one imaging device is positioned at an image plane of a plurality 
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of image planes, where each image plane is displaced in a third dimension with 
respect to each other image plane, the method comprising the steps of: 

orienting the plurality of imaging devices such that the imaging 
devices observe an image object in an image object plane; 
5 using the plurality of imaging devices, obtaining a plurality of 

images of the image object, each image of the plurality of images obtained 
in a respective imaging device; 

converting the plurality of images into a plurality of electronic 
image representations; and 
10 storing the plurality of electronic image representations. 

22. The method of Claim 21, wherein the plurality of imaging devices 
includes a plurality of optical conductors, each optical conductor having a first end 
face positioned at a respective image plane, the step of orienting including 
positioning the first end faces with respect to the image object plane such that each 

15 image plane is substantially parallel to and vertically displaced from the image 
object plane. 

23. The method of Claim 21 , wherein the plurality of imaging devices 
includes a plurality of photo electric devices disposed in an elongate array having 
a longitudinal dimension, the step of orienting including disposing the elongate 

20 array of photo electric devices such that an extension of the longitudinal dimension 
intersects the image object plane at a non-orthogonal angle. 

24. An autofocus apparatus for autofocusing a microscope having a 
focus mechanism, the apparatus comprising: 

a plurality of the imaging devices, at least one imaging device of 
25 the plurality of imaging devices positioned at each respective image plane 
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of a plurality of image planes, where each image plane is displaced in a 
third dimension with respect to each other image plane; 

an image forming apparatus coupled to the plurality of imaging 
devices to form a plurality of electronic image representations, each 
5 electronic image representation representing an image at a respective two- 

dimensional image plane of the plurality of image planes; 

a processor coupled to the image forming apparatus for generating 
a focus location signal by applying a measurement function to the 
electronic image representations; and 
10 means for causing the focus mechanism to adjust the focus of the 

microscope in response to the focus location signal. 

25 . The autofocus of Claim 24, wherein the plurality of imaging devices 
includes a plurality of optical conductors, each optical conductor having a first end 
face positioned at a respective image plane and a second end face coupled to the 

15 image forming apparatus. 

26. The autofocus apparatus of Claim 24, wherein the image forming 
apparatus includes a plurality of charge-coupled device (CCD) cameras. 

27. The autofocus apparatus of Claim 26, wherein the plurality of 
imaging devices includes a plurality of optical conductors, each optical conductor 

20 having a first end face positioned at a respective image plane of the plurality of 
image planes and a second end face coupled to a respective CCD camera. 

28. The autofocus apparatus of Claim 24, wherein the plurality of 
imaging devices includes an elongate array of photo electric devices. 
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29. The autofocus of Claim 28, wherein the elongate array of photo 
electric devices is tilted with respect to an image object plane at which an image 
object located, each image plane of the plurality of image planes being intersected 
by at least one photo electric device of the elongate array of photo electric devices. 

5 30. The autofocus apparatus of Claim 24, wherein the processor 

includes: 

a plurality of image buffers, each image buffer for storing a 
plurality of electronic representations of at least a portion of the image each 
electronic representation representing an image of the portion at a 
10 respective image plane of the plurality of image planes; 

highpass filter means coupled to an image buffer for providing a 
plurality of filtered electronic image representations; and 

means coupled to the high-pass filter means for combining a 
plurality of filtered electronic image representations to produce a focus 
15 metric. 

31. The autofocus apparatus of Claim 21, further including an optical 
filter positioned between the plurality of imaging devices and an image object 
plane. 

32. The autofocus apparatus of Claim 3 1 , wherein the processing means 
20 includes: 

a plurality of image buffers, each image buffer for storing 
a plurality of filtered electronic image representations of at least a 
portion of an image; and 
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means coupled to the plurality of image buffers for 
combining a plurality of filtered electronic image representations to 
produce the optical quality metric. 

33. The apparatus of claim 32, wherein the optical quality metric is 
5 image sharpness. 

34. The autofocus apparatus of Claim 30, wherein the means for 
combining includes: 

means for obtaining augmented intensity values for the filtered 
electronic image representations; and 
10 means for indicating a value of the focus metric for each image 

based upon the augmented intensity values. 

35. A method for autofocus for a microscope having a focus 
mechanism, the method comprising the steps of: 

acquiring a plurality of electronic image representations of an image 
object on an image object plane, wherein each electronic image 
representation represents an image of the image object at a respective 
image plane of a plurality of image planes in which each image plane is 
displaced in a third dimension with respect to every other image plane; 

applying the plurality of electronic image representations to a 
highpass filter to obtain a plurality of filtered electronic image 
representations; 

applying a measurement function to the plurality if filtered 
electronic image representations to obtain a plurality of image sharpness 
values; 



15 
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combining the image sharpness values by relating the image 
sharpness values to image plane positions; 

selecting an image plane position related in the combining step to 
a high image sharpness value; and 

adjusting focus of the microscope based on the selected image plane 
position. 
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